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!= (54) Title: IMAGE SEGMENTING TO ENABLE ELECTRONIC SHOPPING FOR WEARABLE GOODS AND COSMETIC 
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(57) Abstract: A system and corresponding method for segmenting into 
regions a digital image derived from a photograph of pan of a body, such 
as a photograph of the face of a person. The segmented regions can then be 
used as the basis for showing for example the face of the person wearing 
different cosmetic materials, since particular kinds of cosmetic materials 
are applied to particular regions of the face. The image segmenting is 
performed using a segment locator, responsive to a digital image of some 
part of a body and also to generic locating guidance. The segment lo- 
cator provides segment location information that is approximate, and is 
used by a segment delineator to provide a precisely segmented image, in 
the form of at least one binary mask identifying a particular region along 
with the original image. In one embodiment, a template deforming mod- 
ule is used with a stecrable filter module, the generic locating guidance 
is a deformable template (a generic representation of the part of the body 
whose image is being segmented) and associated constraints (indicating 
how the part of the body whose image is being segmented may change 
from individual to individual), and the segment locating information is a 
deformed template. In another embodiment, the image segmenter is part 
of a shopping system allowing a customer to electronically try on wear- 
able goods (including cosmetic materials) or electronically try out cos- 
metic services. The shopping system according to the invention includes: 
a means for providing the digital image derived from a photograph of a 
person; and a remote electronic store responsive to the digital image, the 
remote electronic store including the segmenting system, the remote elec- 
tronic store for providing the customer via a display device with madeup 
images showing how the customer would appear wearing the wearable 
goods or as a result of the cosmetic services. 
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IMAGE SEGMENTING TO ENABLE ELECTRONIC SHOPPING FOR 
WEARABLE GOODS AND COSMETIC SERVICES 

Field of the Invention 

The present invention pertains to the field of shopping 
for goods that are worn by a customer (including for example 
clothes, jewelry and cosmetics) or cosmetic services 
(including for example hair styles and hair treatments) , 
using a display device to allow a shopper to electronically 
(virtually) try on the goods or (virtually) try out the 
cosmetic services. More specifically, the present invention 
is concerned with image segmenting technology used to 
identify segments of an image of a customer where goods would 
be worn or applied (in the case of cosmetics) , or where 
cosmetic services would be directed, thus making it possible 
for a computer to use the segmented image to show the 
customer with different goods worn or applied or to show the 
customer as the customer would appear as a result of cosmetic 
services. 

Background of the Invention 

In purchasing goods to be worn or applied or in 
purchasing cosmetic services, a customer would like to see 
how the customer would look to others when wearing the goods, 
or for example after a makeover (including the use of 
cosmetic materials, hair treatments and hair styling, and 
jewelry) . For example, in purchasing a cosmetic material, it 
is very often useful to try on the cosmetic material, such as 
lipstick, eyeshadow, or blush, before purchasing it. In 
trying on first one cosmetic material and then another, it is 
necessary for the customer to remove the first cosmetic 
material. In addition, to actually try on a cosmetic 
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material, a customer must visit a store selling the cosmetic 
material . 

Today, because of the increased computing capabilities 
of readily available computers, including so-called personal 
computers/ it is reasonable to attempt to have a computer 
provide a customer with an image of the customer's own face 
or an image of more of the customer, modified to illustrate 
how the customer would appear wearing a cosmetic material or 
other wearable goods, including even a hairstyle. To do so, 
however, requires a precise identification in the image of 
those regions of the customer's face or other parts of the 
customer where the goods would be worn (or, in the case of 
cosmetics, where the cosmetic materials would be applied) . 

If this could be done, then a customer could visit a 
retail store and use a computer at the retail store to do a 
virtual w trying on" of the wearable goods, or a virtual 
"trying out" of cosmetic services, and could even do the 
virtual "trying on" or "trying out" from home, or from some 
other, convenient location. 

Summary Of The Invention 

Accordingly, the present invention provides an image 
segmenter and a corresponding method for segmenting an image, 
for segmenting into predetermined regions a digital image 
derived from a photograph of a person or animal, including: a 
segment locator, responsive to the digital image and to 
generic locating guidance, for providing segment location 
information; and a segment delineator, responsive to the 
digital image and to the segment location information, for 
providing a segmented image including a binary mask. 

In a further aspect of the invention, the segment 
locator is a template deforming module, and the segment 
locator is a steerable filter module. In such an embodiment, 
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the generic locating guidance is a deformable template (a 
kind of generic representation of the part of the body whose 
image is being segmented) and associated constraints 
(indicating how the part of the body whose image is being 
segmented may change from individual to individual) , and the 
segment locating information is a deformed template. 

In other further aspects of the invention, either the 
segment locator or the segment delineator is based on a 
neural network, or either the segment locator or the segment 
delineator is based on principal component analysis, or 
either the segment locator or the segment delineator is based 
on wavelet graph analysis. 

In yet other. further aspects of the invention, the 
predetermined regions include a lipstick region, an eyeshadow 
region, and a blush region. The pre -determined regions can 
in general be any regions of a body, including regions where 
eyeglasses are worn, or where a shirt or blouse is worn, or 
where pants are worn, or the individual's hair (for helping 
show a hair style or hair treatment) . 

In yet another aspect of the invention, the image 
segmenter is part of a shopping system allowing a customer to 
electronically try on wearable goods (including cosmetic 
materials) or electronically try out cosmetic services. The 
shopping system according to the invention includes: a means 
for providing the digital image derived from a photograph of 
a person; and a remote electronic store, responsive to the 
digital image, the remote electronic store including the 
segmenting system, the remote electronic store for providing 
the customer via a display device with madeup images showing 
how the customer would appear wearing the wearable goods or 
as a result of the cosmetic services. 

In another aspect of the invention, a method is provided 
for enabling a customer to electronically try on wearable 
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goods (including cosmetic materials) or electronically try 
out cosmetic services, including the steps of: taking a 
photograph of the customer and providing a digital encoding 
of the photograph; and performing a two-part process to 
precisely identify pre-determined regions where the wearable 
goods would be worn or applied or where the cosmetic services 
would be directed, a first part for approximately locating 
each of the regions, and a second part to delineate the 
approximately located regions. 

Brief Description of the Drawings 

The above and other objects, features and advantages of 
the invention will become apparent from a consideration of 
the subsequent detailed description presented in connection 
with accompanying drawings, in which: 

Fig. 1 is a flowchart/ block diagram showing the 
elements of a system for electronically trying on 
cosmetics, as an example of wearable goods, allowing for 
the purchase of the cosmetics tried on, according to the 
present invention; 

Fig. 2 is an illustration of a human face, indicating 
regions identified by the present invention where different 
types of cosmetics (eyeshadow, blush, and lipstick) would 
be applied; 

Fig. 3 is a flow chart showing the steps according to 
the present invention for electronically trying on and 
ordering cosmetic materials; 

Fig. 4 is a flowchart/ block diagram showing the 
principal components of a face segmenter (one kind of image 
segmenter) , and in particular a face segmenter for 
identifying where on a human face to apply different types 
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of cosmetic materials; 

Fig. 5 is a flow chart showing one embodiment of a 
method for providing a segmented image, according to the 
present invention; and 

Fig. 6 is a flowchart/ block diagram showing the 
principal components of an image segmenting system 
according to the present invention. 

Best Mods For Carrying Out The Invention 

The image segmenting invention of the present invention 
will first be described with respect to identifying segments 
(regions) in a photograph of a customer showing where 
cosmetic materials would be applied. Such image segmenting 
is. called here face segmenting in case of a photograph of 
only a customer's face. Then the preferred embodiment of the 
image segmenting invention itself will be described, again in 
particular with respect to face segmenting allowing a 
customer to electronically try on cosmetics. Finally, other 
embodiments of the image segmenting of the present invention 
will be described, and also other applications, besides 
electronically trying on cosmetics. 

Face (Image) Segmenting to Allow Electronically Trying On 
Cosmetics 

Referring now to Fig. 1, a system enabling a customer 
12' to electronically try on and possibly order cosmetic 
materials, such as eyeshadow, lipstick, or blush, includes 
one or more kiosks 10, which are located in convenient 
locations, such as in retail stores, shopping malls or in 
specialty stores, or in even as standalone facilities in 
parking lots and on street corners. In addition, the system 
includes a remote electronic store 20, which interfaces with 
the kiosk 10 via respective interfaces 24 and 13 . The system 
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optionally includes a personal computer 14 located in a 
cosmetics retail location 18, or even in the kiosk 10. 
Alternatively, a customer 12/ can use any personal computer 
14, such as a home personal computer, to interface with the 
system of the present invention. In the preferred 
embodiment, a customer 12' uses a personal computer from a 
convenient location, such as the customer's home, to 
interface with the system of the present invention. 

Still referring to Fig. 1, according to the present 
invention, a customer 12 visits the kiosk 10 where a 
photograph of the customer is taken, possibly by an 
assistant, but preferably in a purely automated process (for 
example by a machine like those that automatically take 
photographs of tourists at pavilions or other locations 
visited mostly by tourists) . The photograph is provided to 
the remote electronic store 20 as a digitized image, along 
with the customer's password. 

It is of course not necessary for a customer to visit a 
kiosk to have the customer's photograph taken. If the 
customer has an appropriate camera (high-resolution, color) 
and equipment needed to provide a photograph as a digital 
image (equipment that may be integral with a camera, such as 
a. digital camera) , the customer could load a digital image of 
a photograph of the customer onto a home computer and provide 
the digital image to the remote electronic store 20 over the 
Internet . 

The remote electronic store includes an image maker 21 
that examines the image, as will be described below, to 
determine what regions of the image correspond to regions of 
the customer's face where different types of cosmetics would 
be applied and to produce a set of (binary) masks (one for 
each region) that can overlay the image and show the customer 
how the customer would appear wearing different cosmetic 
materials. The combination of the set of binary masks along 
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with the original image is here called a segmented image. 
The image maker 21 saves the segmented image and password in 
a customer database 25. Later, the customer 12 1 uses a 
personal computer 14 to access the remote electronic store 20 
so as to be able to electronically try on different cosmetic 
materials, such as eye shadow, lipstick, and blush. The 
customer provides the customer's password and indicates 
through choice indicators what cosmetic materials the 
customer would like to "try on." (The customer would also 
use choice indicators to place an order,) The image maker 
then retrieves from the customer database the segmented image 
based on the customer password and retrieves from a cosmetic 
database 22 information about the cosmetics selected for 
electronically ' (virtually) "trying on" by the customer, and 
provides to the customer an image, called here a made-up 
image, of a customer wearing the selected cosmetic materials. 
If the customer 12 1 then chooses to purchase the cosmetic 
material, the customer can provide a credit card number or 
other billing information to the remote electronic store 20, 
and the selected cosmetic materials and credit card number 
are then stored in a cosmetics retail database 16 as an order 
to be processed. An interface 17 with a retail location then 
provides the order to an appropriate cosmetics retail 
location 18, a location where the customer might conveniently 
pick up and possibly pay for the order. Alternatively, 
instead of having the customer visit the cosmetics retail 
location, the cosmetics retail location 18 may ship the 
ordered cosmetic materials to the customer. 

Still referring to Fig. 1, the kiosk 10 includes a turn- 
key photo system 11, which includes a camera 15 capable of 
taking a high resolution color photograph of a customer's 
face. A customer 12 then uses the interface 13 with the 
remote electronic store 20 to obtain a password. In addition 
the customer can provide for the first time an indication of 
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what types of cosmetics the customer is interested in. This 
preference information is called here cosmetics selection 
data. It can indicate particular preferred cosmetics (by- 
brand name) , or can indicate information useful to the remote 
electronic store for suggesting particular cosmetics. For 
example, the customer can be asked to indicate whether the 
customer plays sports, or is often outdoors, which would 
indicate that some kinds of cosmetics are more suitable then 
others . 

The kiosk 10 provides to the electronic store, and in 
particular to the image maker 21 through the interface with 
the kiosk, the customer ! s password, image data and cosmetic 
selection data. Later, when the customer 12 1 uses the 
personal computer 14 to electronically try on cosmetic 
materials, the remote electronic store 20 suggests particular 
cosmetic materials to the customer. To suggest cosmetic 
materials, the customer interface 23 pulls- out of the 
customer database 25 the cosmetics selections data provided 
by the customer at the kiosk, analyzes the cosmetics 
selection data and refers to the cosmetics data base 22 for 
cosmetics product information, indicated as cosmetics data in 
Fig. 1, based on the cosmetics selection data. In the 
preferred embodiment, cosmetics data includes information not 
only about the product itself such as its ingredients, but 
also data useful to the image maker in determining how to 
display cosmetic material when it is applied to a human face. 
The cosmetics data is retrieved from the cosmetics database 
22 by the image maker 21 in preparing a made-up image. It is 
of course also sometimes advantageous to have a customer 
input further cosmetics selection data through the personal 
computer 14 while in the course of electronically trying-on 
cosmetic materials. 

In the preferred embodiment, the customer interface 23 
also suggests cosmetics materials based on the original 
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(color) image of customer, stored with the segmented image in 
the customer database 25, whether or not the customer ever 
provides cosmetics selection data. 

Fig. 2 illustrates different regions of a face the image 
maker 21 (Fig. 1) must identify, regions where different 
kinds of cosmetic materials would be applied. For example, 
the image (digitized version of the original photograph taken 
at the kiosk) will include a region 31 where lipstick would 
be applied. The image maker might also identify a region 
(not shown) where lipliner would be applied. In addition, 
the image would include regions 32a-b where blush would be 
applied. Finally, an image would include regions where eye 
cosmetics would be applied, including for example eyeliner 
(in regions 33a-b and 35a-b) and eyeshadow (in regions 34a- 
b) . In addition, the image maker would identify regions (not 
shown) where mascara would be applied (on the eyelashes) . 

Referring now to Fig. 3, a flow chart is shown 
indicating the overall process beginning with the customer 
visiting a kiosk and terminating in a customer ordering 
cosmetics . 

Face (Image) Segmenting Using a Deformable Template and a 
Steerable Filter Module 

The image segmenting of the present invention, referred 
to as face segmenting in case of identifying only segments 
(regions) in an image of a face, will now be described in the 
particular case of identifying segments of a human face where 
different cosmetic materials would be applied. 

Referring now to Fig, 4, a face segment er 40, a sub- 
system of the image maker 21 used to identify where in an 
image (digitized photograph of the customer's face) different 
cosmetic materials would be applied, is shown in the 
preferred embodiment as including a template deforming module 
41 and a steerable filter module 42. The template deforming 
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module 41 uses as an input the image of the customer acquired 
at the kiosk 10 and also uses a deformable template for a 
human face, along with constraints on how the human face can 
change, from one person to another. For example, a human 
face will have a typical range of aspect ratios of height to 
width, so that in varying for example the height, the width 
can be assumed to be known to within some typical range of 
values. The template deforming module 41 produces a deformed 
template which provides an approximate location of the target 
regions, i.e.. the regions where the different cosmetics 
materials would be applied. 

In the preferred embodiment, in producing the deformed 
template, the template deforming module 41 uses as input not 
only the original image, but also key points marked on the 
image by a technician or even by the customer. The key 
points provide the precise locations of some particular 
features of the face in the image, such as the center of the 
pupils of the customer. The deformable template deforms the 
starting template by fixing the corresponding locations in 
the deformable template to the precisely located key points 
in the image, and as the template is deformed in one 
dimension to fit the key points, it also deforms in the 
orthogonal direction according to the constraint information, 
absent some overriding information. 

The steerable filter module 42 then uses the deformed 
template and the original image to more precisely locate the 
target regions. It does so by essentially identifying for 
each particular region edges in the image that delineate the 
region, using the deformed template merely as a guide to 
locating the different regions. The final product of the 
steerable filter module is a segmented image, i.e. the 
original image and in addition a set of binary masks (one for 
each region) that are to be overlaid on the original image 
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and that include information about where on the image 
different cosmetic materials would be applied. 

In the preferred embodiment, the original image is 
marked (either by a technician or by the customer) to locate 
the key points either at the kiosk 10 (see Fig. 1) or at the 
remote electronic store. Instead of manually locating key 
points, however, it is also possible to use other technology, 
such as neural networks, to automatically examine the image 
provided by the high resolution camera 15 (Fig. 1) so as to 
precisely locate the key points. 

Referring now to Fig. 5, a flow chart is provided 
indicating the process performed by the image maker 21 (Fig. 
1) . In the preferred embodiment the image that is acquired 
at the kiosk and used as an input to the image maker is a 24- 
bit color, high-resolution image, along with a set of co- 
ordinates locating on the image the key points, which in the 
preferred embodiment include the center of the iris of each 
eye, the points making up a vertical line drawn down the 
center of the face, and the points making up a horizontal 
line drawn across the corners of the mouth. 

Other Image Segmenting and Other Applications 

Besides using a deformable template coupled with a 
steerable filter module, it is sometimes advantageous to use 
various other methods for providing a segmented image, 
including principal component analysis (based on modeling 
images as reducible matrices in a high-dimensional space) , 
wavelet graph analysis (similar to deformable template 
matching, except that Gabor wavelet transform localization is 
included in every node in the deformable template while the 
deformable template is being adjusted to fit the image) , and 
neural network based recognition system (for example those 
using neural networks trained with a series of customer 
images using back propagation to determine the weights for 
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connections between radial basis functions in a network of 
such functions) . According to the present invention, 
whatever particular methods are used, image segmenting is a 
task bifurcated into two subtasks: finding the general 
location of a segment (or equivalent ly approximately 
determining a region), and then delineating the segment (i.e. 
more precisely determining the region) . 

Referring now to Fig. 6, an image segmenter 60 is shown 
as including a segment locator module 61 for locating a 
segment generally and a segment delineator module 62 for 
delineating the approximately defined and located segment. 
The segment locator 61 uses as input the image derived from 
the photograph of a person (or even a non-human animal) , as 
well as generic guidance on locating one or more segments in 
the image, i.e. guidance that is useful for an image of the 
same part of any body, not guidance associated with a 
particular body. Thus, for example, in the case of the image 
segmenter of Fig. 4 (called a face segmenter when applied to 
segmenting the image of a face, compared to other parts of a 
body) , the generic locating guidance includes a (generic) 
deformable template and (generic) constraints. 

The segment locator 61 may optionally use key points to 
assist it in its subtask, as in the embodiment of a face 
segmenter shown in Fig. 3. As mentioned above, however, the 
need for key points can be eliminated by adjusting the 
segment locator to accomplish its subtask without reliance on 
key points- Also, if key points are used, they can be 
provided automatically by a pre -processor stage (not shown) , 
such as a neural network based system for identifying in an 
unmarked image key points of use to the segment locator 61. 

The output of the segment locator 61 is segment location 
information, such as is provided by the deformed template in 
the face segmenting embodiment of Fig. 4. The segment 
delineator 62 uses the segment location information along 
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with the original image, unmarked with key points, to more 
precisely define the segment, ultimately providing a 
segmented image, i.e. a set of binary masks along with the 
original unmarked image. 

Besides using a deformable template for the segment 
locator module 61 and a steerable filter module for the 
segment delineator module 62, as in the face segmenting 
embodiment of Fig. 4, each module can be based on a different 
neural network, one trained to recognize the general location 
of predetermined target regions (such as regions associated 
with the eyes or lips) , and one trained to precisely 
discriminate between a target region and its surroundings. 

As has been mentioned above, the present invention is 
intended to comprehend applications involving the segmenting 
into regions of the image of any part of a body, and not 
necessarily the body of a human. In addition, an image 
segmented according to the present invention can be used not 
only to display how cosmetics would appear on a wearer, but 
also how other items would appear, including jewelry, clothes 
(including a hat) , eyeglasses, a hairstyle and treated hair, 
or even a tattoo or other form of wearable art . The present 
invention allows electronically trying on any wearable goods 
and trying out cosmetic services, allowing shopping for such 
goods and services even from home over the Internet . 

It is to be understood that the above -described 
arrangements are only illustrative of the application of the 
principles of the present invention. Numerous modifications 
and alternative arrangements may be devised by those skilled 
in the art without departing from the spirit and scope of the 
present invention, and the appended claims are intended to 
cover such modifications and arrangements. 
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What is claimed is: 



1. An image segment er for segmenting into predetermined 
regions a digital image derived from a photograph of a 
person or animal, comprising: 

a) a segment locator, responsive to the digital 
image and to generic locating guidance, for providing 
segment location information; and 

b) a segment delineator, responsive to the digital 
image and to the segment location information, for 
providing a segmented image including a binary mask. 

2. An image segmenter as in claim 1, wherein the segment 
locator is a template deforming module, wherein the 
segment locator is a steerable filter module, wherein 
the generic locating guidance is a deformable template 
and associated constraints, and further wherein the 
segment locating information is a deformed template. 

3. An image segmenter as in claim 1, wherein either the 
segment locator or the segment delineator is based on a 
neural network. 



4. An image segmenter as in claim 1, wherein either the 
segment locator or the segment delineator is based on 
principal component analysis. 

5. An image segmenter as in claim 1, wherein either the 
segment locator or the segment delineator is based on 
wavelet graph analysis. 

6. An image segmenter as in claim 1, wherein the 
predetermined regions include a lipstick region, an 
eyeshadow region, and a blush region. 
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7. An image segmenter as in claim 1, wherein the image 
segment er is part of a shopping system allowing a 
customer to electronically try on wearable goods or 
electronically try out cosmetic services, the shopping 
system comprising: 

a) a means for providing the digital image derived 
from a photograph of a person; and 

b) a remote electronic store, responsive to the 
digital image, the remote electronic store including the 
segmenting system, the remote electronic store for 
providing the customer via a display device with madeup 
images showing how the customer would appear wearing the 
wearable goods or as a result of the cosmetic services. 

8. An image segmenter as in claim 7, wherein the remote 
electronic store suggests particular wearable goods or 
cosmetic services based on the image or based on 
selection data provided by the customer. 

9. A method for enabling a customer to electronically try 
on wearable goods or electronically try out cosmetic 
services, comprising the steps of: 

a) taking a photograph of the customer and' 
providing a digital encoding of the photograph; and 

b) performing a two-part process to precisely 
identify pre -determined regions where the wearable goods 
would be worn or applied or where the cosmetic services 
would be directed, a first part for approximately 
locating each of the regions, and a second part to 
delineate the approximately located regions. 
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10. The method of claim 9, wherein the first part uses a 
template deforming module and the second part uses a 
steerable filter module. 

11. The method of claim 9, wherein at least one of the two 
parts uses a neural network. 

12. The method of claim 9, wherein at least one of the two 
parts uses principal component analysis. 

13. The method of claim 9, wherein at least one of the two 
parts uses wavelet ' graph analysis. 

14. A method as in claim 9, further comprising the steps of: 

a) obtaining from the customer at least some 
wearable goods or cosmetics services selection data; and 

b) recommending to the customer particular 
wearable goods or cosmetics services based on the 
wearable goods or cosmetics services selection data or 
based on the customer image. 

15. A method as in claim 14, wherein the wearable goods or 
cosmetics services selection data includes lifestyle 
information, such as whether the customer often 
participates in sporting activities. 
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Customer visits a Kiosk to have high 
resolution image made and to 
indicate cosmetics selection data 
(including lifestyle information), and 
receives password and contact 
information, i.e. how to access 
Remote Electronic Store (RES). 
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Kiosk provides image and cosmetics 
selection data to RES. . 
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Customer connects personal 
computer to RES using password. 
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RES provides a menu and asks for 
customer choice indicator. 
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Customer views one or more 
cosmetics from the subset of 
cosmetics 
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Customer selects to "try on" a 
particular brand of lipstick, blush, 
and eyeshadow. 
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Customer examines madeup image 

(i.e. electronically tries on the 
cosmetics) and inspects cosmetics 
product information (ingredients, 
pricing, and notices). 
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Apply given key points in image to corresponding points on template, thereby 
deforming template so that it approximates face in image. 



Use steerable filters to find a parabolic curve segment that approximates the edge 

of the upper lip. 

i 

Look in the area immediately below the parabolic curve to determine the 
approximate color of the lips, and then use the color to determine whether the 
mouth in the image is open. (Look for non-lip colored regions a short distance 
down from the top of the upper lip.) 



± 

Use the lip color to find a parabolic curve segment that approximates the edge of 
the lower lip. If the mouth is open, use the lip color to approximate the inner 

boundaries of the lips. 



Use steerable filters to trace around the boundary of the lips. If the tracing path 
strays too far from the approximation, reject that tracing segment. 



Analyze the loop tracings to locate the two peaks of the top edge of the upper lip. 



i 

Use the color data to further refine the contour of the lip edges, by finding points 
on the boundary between lip-colored regions an non-lip-colored regions. 



Smooth the lip edges by fitting multiple parabolic curve segments to the refined 
edges. Select the areas between the appropriate lip edges as the lip 

segmentation. 
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